The Vickers hardness test is widely used for metal materials. The hardness given by the test is inversely proportional to the ratio of the test force to the indentation area. Since it is inconvenient to measure the area of the indentation manually, it is usually approximated by measuring its diagonal length. On this basis, we propose a new indentation segmentation network based on existing work. Taking advantage of its high segmentation accuracy, we use the indentation area to calculate the Vickers hardness directly, rather than relying on the measurement of the diagonal length. Experimental results show that the indentation areas obtained by this method are well below the International Standardization Organization (ISO) standard and very close to those from manual measurements. In some cases, this method can be a good alternative to manual measurements.